Methods and systems for augmenting visual content

ABSTRACT

Systems and methods are described for augmenting visual content with a sponsored object instead of a selected object. An illustrative method receives an input selecting an object for augmenting visual content, determines whether a property of the selected object matches a property of a sponsored object included in a database of sponsored objects, and in response to determining that the property of the selected object matches the property of the sponsored object, augments the visual content with the sponsored object.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage application under 35 U.S.C. § 371of International Application PCT/US2018/059753, filed Nov. 8, 2018,which is hereby incorporated by reference herein in its entirety.

BACKGROUND

The present disclosure relates to augmenting visual content, and moreparticularly, to methods and systems for generating for displaysponsored augmented reality objects on a user display device in realtime or in post-processing.

SUMMARY

Augmented reality (AR) allows a user to view supplemental contentsuperimposed onto an image or view of the real world. For example, inembodiments where the user views an image of the real world on an ARdisplay (such as a smartphone), the supplemental content is overlaid ontop of the real-world image as to appear to be part of the real-worldimage. In other embodiments, where the user views the real world throughan AR display (such as AR glasses), the supplemental content isdisplayed in a portion of the AR display where the user would otherwisesee a part of the real world. Such supplemental content enhances theuser experience and allows the user to interact with and/or customizethe real-world image to their liking. For example, the user can viewand/or capture images of the real world with user-selected supplementalcontent superimposed thereon and/or integrated therein.

This creates an opportunity for application developers to generaterevenue from marketers wishing to promote products to the user of the ARdisplay. For example, a common method by which application developersprovide free-to-use applications to users is by making the applications“ad-supported,” meaning the developers generate income by advertisementsplaced in the applications instead of the users having to pay to use theapplications. However, users often do not like unsolicited or intrusiveadvertisements appearing in applications. Conventional banneradvertisements or pop-up advertisements can be especially bothersome tousers of AR displays who want to view images of the real world that arebeing obscured by intrusive advertisements. There exists, therefore, aneed for a way to insert advertisements or other revenue-generatingcontent into images displayed on AR displays that is non-intrusive, andthus preferable, to users of the AR displays.

Accordingly, to overcome such problems, methods and systems aredisclosed herein for augmenting visual content with promotional contentthat is both non-intrusive and specifically tailored to a user. Inparticular, the embodiments described herein provide for substitutingsponsored objects for generic objects selected by a user when augmentingvisual content. The sponsored objects may be branded and/or designed toaccurately represent products promoted by marketers, thereby allowingthe marketers to insert advertisements into the AR content requested bythe user, and because the sponsored object is substituted for a genericobject selected by the user, the advertisement is both non-intrusive (itis, after all, an object requested by the user) and specificallytailored to the user since the user requested that particular object tobe placed in the AR content.

In one illustrative embodiment, a real-world image is displayed on an ARdevice, such as a smartphone. The real-world image may include a person.The AR device receives a selection of an object with which to augmentthe real-world image. For example, the AR device displays a list ofobjects with which the real-world image can be augmented, and a user mayselect a pair of generic aviator glasses from the list to add to theimage of the person. Upon receiving the selection of the object (e.g.,the generic aviator glasses), the AR device queries a database ofsponsored objects for a sponsored object matching a property of theselected object. For example, a marketer may be promoting aviatorglasses of a particular brand, and when the AR device queries thedatabase of sponsored objects for aviator glasses, the database returnsthe sponsored aviator glasses. The AR device then augments thereal-world image with the sponsored aviator glasses instead of thegeneric aviator glasses selected by the user. For example, the AR devicedisplays the sponsored aviator glasses overlaid onto the image of theperson. The AR device may further store the augmented real-world image.This allows the marketer to insert an advertisement (e.g., the sponsoredaviator glasses) into augmented content generated based on a selectionprovided by the user and in a way that is non-intrusive to the user.

In another illustrative embodiment, a real-world environment is viewedthrough an AR device, such as AR glasses. The real-world environment mayinclude a house with a driveway. The AR device receives a voice commandfrom a user to show a sports car in the house's driveway. A marketer maybe promoting a particular sports car, and upon receiving the selection,the AR device queries a database of sponsored objects for a sports car.The database then returns the sponsored sports car, whereafter the ARdevice substitutes the sponsored sports car for a generic sports car,and generates a display of the sponsored sports car such that when auser views the real-world environment through the AR glasses, thesponsored sports car appears in the house's driveway.

The present disclosure further describes methods and systems forreceiving input specifying a target position where the object should beplaced in the augmented content. For example, the user may specify thatthe aviator glasses should be placed on the face of the person in theimage, or that the sports car should be placed in the house's driveway.In other embodiments, the methods and systems may automaticallydetermine where to place the object in the augmented content. Forexample, the AR device may identify the face of the person in the image,retrieve a template of the aviator glasses from the database, andgenerate a display of the aviator glasses with a size relative to thesize of the face of the person in the image, and at an appropriateposition in the image (e.g., over the eyes and/or resting on the nose ofthe person in the image). In some embodiments, the appropriate positionin which to place the object in the image may be determined based on oneor more anchor points. For example, the nose and/or eyes of the personin the image may serve as anchor points for the aviator glasses, and maybe correlated with corresponding anchor points included in the templateof the aviator glasses, so as to guide placement and/or sizing of theaviator glasses. In still further embodiments, the methods and systemsdetermine a position of a light source illuminating the real-worldimage. For example, the AR device determines a direction from which thesun is shining on the house in the real-world environment. The AR devicethen adjusts a visual characteristic (e.g., a shadow) of the sports carbased on the direction from which the sun is shining on the house, suchthat the shadow generated by the sports car is appropriate for thereal-world environment.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects and advantages of the disclosure will beapparent upon consideration of the following detailed description, takenin conjunction with the accompanying drawings, in which like referencecharacters refer to like parts throughout, and in which:

FIG. 1 shows an illustrative example of a user interface for augmentingvisual content with a sponsored object, in accordance with someembodiments of the disclosure;

FIG. 2A shows an illustrative system for augmenting visual content witha sponsored object, in accordance with some embodiments of thedisclosure;

FIG. 2B shows an illustrative scenario for augmenting visual contentwith a sponsored object, in accordance with some embodiments of thedisclosure;

FIG. 3 shows a diagram of illustrative devices of the system of FIG. 2A,in accordance with some embodiments of the disclosure;

FIG. 4 is a flowchart of an illustrative process for augmenting visualcontent with a sponsored object, in accordance with some embodiments ofthe disclosure;

FIG. 5 is a flowchart of a detailed illustrative process for generatingaugmented visual content, in accordance with some embodiments of thedisclosure;

FIG. 6 is a flowchart of a detailed illustrative process for determiningwhether a property of a selected object matches a property of asponsored object, in accordance with some embodiments of the disclosure;

FIG. 7 is a flowchart of a detailed illustrative process for positioninga sponsored object in visual content, in accordance with someembodiments of the disclosure;

FIG. 8 is a flowchart of another detailed illustrative process forpositioning a sponsored object in visual content, in accordance withsome embodiments of the disclosure;

FIG. 9 is a flowchart of a detailed illustrative process for modifying asponsored object based on a light source illuminating visual content, inaccordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

FIG. 1 shows an illustrative example of a user interface for augmentingvisual content with a sponsored object, in accordance with someembodiments of the disclosure. In particular, FIG. 1 shows a scenario100 where an augmented reality (AR) device 102 displays visual content104, including a subject object 106, that has been augmented with asponsored object 108 having a branding 110. AR device 102 may be anydevice capable of displaying visual and/or augmented content, such as asmartphone, tablet computer, AR glasses, AR head-mounted display (HMD),virtual reality (VR) HMD, or any other user equipment usable to displayvisual content 104, and/or combination of the same.

In that regard, visual content 104 is a live video or otherlive-captured moving image, three-dimensional image, panoramic or otherimmersive image (e.g., a 360-degree image or series of live imagessurrounding a particular viewpoint), live VR or AR renderings, and/orany other live-captured visual media or combination of the same. Forexample, visual content 104 may be a live-capture or live-stream ofimages received from one or more lenses or image-capturing devices.

Subject object 106 may be any feature or element included in visualcontent 104. Subject object 106 may be the focus of visual content 104(e.g., the visual content may be an image of a person, as shown in FIG.1), or may be one of a plurality of features or elements included in thevisual content. Sponsored object 108 may correspond to a generic objectthat a user has selected for augmenting visual content 104, as furtherdescribed below. In some embodiments, sponsored object 108 represents aproduct that is being promoted by a marketer. For example, as shown inFIG. 1, sponsored object 108 is a pair of aviator glasses with branding110 “RAY-BAN” included thereon. In an illustrative embodiment describedbelow, the user selects a generic pair of aviator sunglasses with whichto augment visual content 104, and AR device 102 instead augments visualcontent 104 with sponsored object 108 (e.g., the RAY-BAN aviatorglasses).

FIG. 2A shows an illustrative system for augmenting visual content. Inparticular, FIG. 2A shows a system 200 where AR device 102 displaysvisual content 104. Visual content 104 may be captured by AR device 102,or may be received from a content source 210 via a communication link232. Content source 210 may be any server or storage device from whichvisual content 104 may be retrieved by AR device 102. The communicationlink 232 may be any network or communication equipment or medium bymeans of which visual content 104 can be retrieved from content source210. In one illustrative example, the communication link 232 includesdevices connected via the Internet.

One or more applications for augmenting visual content may beimplemented on any one or a combination of AR device 102, content source210, and/or an application server 220, each of which may function as astand-alone device or may be part of a network of devices. Variousnetwork configurations of devices may be implemented and are discussedin more detail below. In the system 200, there may be multiple ARdevices 102, but only one is shown in FIG. 2 to avoid overcomplicatingthe drawing.

AR device 102 may be coupled to a communication network 230.Communication network 230 may be one or more networks including theInternet, a mobile phone network, mobile voice or data network (e.g., a4G or LTE network), cable network, public switched telephone network, orother types of communication network or combinations of communicationnetworks. Content source 210, application server 220, and AR device 102may be connected to communication network 230 via one or morecommunication paths 232, such as, a satellite path, a fiber-optic path,a cable path, free-space connections (e.g., for wireless signals), orany other suitable wired or wireless communication path or combinationof such paths.

Although communication paths 232 are not drawn directly between ARdevice 102 and content source 210 or application server 220, thesedevices may communicate directly with each other via communication paths232, such as short-range point-to-point communication paths, such as USBcables, IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared,IEEE 802-11x, etc.), or other short-range communication via wired orwireless paths. BLUETOOTH is a certification mark owned by BluetoothSIG, INC. The devices may also communicate with each other directlythrough an indirect path via communication network 230.

The system 200 may include more than one content source 210, but onlyone is shown in FIG. 2 to avoid overcomplicating the drawing. Inaddition to being a source for visual content 104, content source 210may include one or more databases 212 of sponsored objects 108. Database212 includes one or more tables of sponsored objects 108 and theircorresponding brandings 110. In some embodiments, database 212 includesa table of properties associated with a sponsored object 108. Forexample, as shown in FIG. 2A, database 212 includes a table in which twotypes of sponsored objects (e.g., sunglasses and hats) are listed withvarious properties (e.g., aviators, oversized, angular) and brandings110 (e.g., RAY-BAN, OAKLEY, NIKE, ADIDAS) associated with each. In someembodiments, content source 210 stores a plurality of templates ofsponsored objects 108 associated with each of the sponsored objects 108included in database 212. The templates of the sponsored objects 108 mayinclude design parameters and/or other data enabling AR device 102and/or application server 220 to generate an instance (e.g., arendering) of the associated sponsored object 108. Content source 210may also include different types of content (including visual contentstored by a user), in a location remote from AR device 102.

Metadata may be associated with visual content 104 and/or the selectedor sponsored object 108. The metadata may include indications orproperties of subject object 106, one or more anchor points or featuresof subject object 106, properties of sponsored object 108, userpreferences, user profile information, and/or any other type of datathat enables a computer to match a selected object with a sponsoredobject 108 and/or determine an appropriate size for and position atwhich to place sponsored object 108 in visual content 104.

The augmented content generation application may be, for example, astand-alone application implemented on one or more of the devicesdescribed above. For example, the augmented content generationapplication may be implemented as software or a set of executableinstructions which may be stored in storage 308 (described below withreference to FIG. 3), and executed by control circuitry 304 (describedbelow with reference to FIG. 3) of AR device 102. In some embodiments,the augmented content generation application is a client/serverapplication where only a client application resides on AR device 102,and a server application resides on application server 220. For example,an augmented content generation application may be implemented partiallyas a client application on control circuitry 304 of AR device 102 andpartially on application server 220 as a server application running oncontrol circuitry 304 of application server 220. When executed bycontrol circuitry 304 of application server 220, the augmented contentgeneration application may instruct control circuitry 304 to generatethe augmented content generation application output (e.g., sponsoredobject 108 or the augmented visual content) and transmit the generatedoutput to AR device 102. The server application may instruct controlcircuitry 304 of content source 210 to transmit a template for and/ormetadata associated with sponsored object 108 to application server 220and/or AR device 102. The client application may instruct controlcircuitry 304 of AR device 102 to generate the augmented contentgeneration application output (e.g., sponsored object 108 and/or theaugmented content).

AR device 102 may operate in a cloud computing environment to accesscloud services. In a cloud computing environment, various types ofcomputing services for content sharing, storage or distribution (e.g.,informational sites or social networking sites) are provided by acollection of network-accessible computing and storage resources,referred to as “the cloud.” For example, content source 210 andapplication server 220 provide computing and storage resources to ARdevice 102. Cloud resources may be accessed by AR device 102 using, forexample, a web browser, a desktop application, a mobile application,and/or any combination of access applications of the same. AR device 102may be a cloud client that relies on cloud computing for applicationdelivery, or AR device 102 may have some functionality without access tocloud resources. In some embodiments, the AR device 102 uses cloudresources for processing operations, such as the processing operationsperformed by the processing circuitry 306 of application server 220,described in relation to FIG. 3. Some applications running on AR device102 may be cloud applications, that is, applications delivered as aservice over the Internet, while other applications may be stored andrun on AR device 102. An example of such an embodiment is furtherdescribed below with reference to FIG. 2B.

FIG. 2B shows a scenario wherein visual content is augmented with asponsored object, in accordance with some embodiments of the disclosure.In FIG. 2B, visual content 104 is a real-world view 250 as viewed withan AR device 260. In the example shown in FIG. 2B, real-world view 250includes a house with a driveway, and a user views real-world view 250via AR glasses. The user then provides, and AR device 260 receives, aninstruction (e.g., a voice command) to augment real-world view 250 withan object. For example, the user may provide a voice command to “show asports car in the driveway.”

Upon receiving the instruction, AR device 260 sends a request toapplication server 220 to augment real-world view 250 with the objectselected by the user. Application server 220 identifies a property ofthe selected object (e.g., that the object is a sports car), and queriesdatabase 212 of the content server 210 for a sponsored object 108matching the property of the selected object. The content server 210then returns a matching sponsored object 108 (e.g., a template of asponsored sports car) to application server 220. Upon receivingsponsored object 108, application server 220 generates an instance ofsponsored object 108 based on the template to provide to AR device 260or provides the template of sponsored object 108 to AR device 260.

AR device 260 then generates for display sponsored object 108. Forexample, as shown in FIG. 2B, the AR glasses includes a left view 265 aand a right view 265 b. Left view 265 a matches real-world view 250.Right view 265 b shows real-world view 250 with an instance 267 ofsponsored object 108 overlaid thereon. For example, right view 265 bshows a sports car 267 in the driveway of the house. In someembodiments, both left view 265 a and right view 265 b of AR device 260display instance 267 of sponsored object 108.

FIG. 3 shows a generalized embodiment of illustrative user device 300and server device 320. As depicted, user device 300 may be a smartphoneor tablet, although those skilled in the art will appreciate thatvarious other devices, such as cameras, AR or virtual reality glasses orother head-mounted displays, etc., may be also be used for augmentingvisual content with a sponsored object, and may have the same or similarcomponents to those of user device 300. The user devices 300 may receivecontent and data via an input/output (hereinafter “I/O”) path 302. I/Opath 302 may provide visual content (e.g., Internet content, contentavailable over a local area network (LAN) or wide area network (WAN),and/or other content), objects for augmenting the visual content, anddata to control circuitry 304, which includes processing circuitry 306and storage 308. Control circuitry 304 may be used to send and receivecommands, requests, and other suitable data using I/O path 302. I/O path302 may connect control circuitry 304 (and specifically processingcircuitry 306) to one or more communication paths (described below). I/Ofunctions may be provided by one or more of these communication paths,but are shown as a single path in FIG. 3 to avoid overcomplicating thedrawing.

Control circuitry 304 may be based on any suitable processing circuitrysuch as processing circuitry 306. As referred to herein, processingcircuitry should be understood to mean circuitry based on one or moremicroprocessors, microcontrollers, digital signal processors,programmable logic devices, field-programmable gate arrays (FPGAs),application-specific integrated circuits (ASICs), etc., and may includea multi-core processor (e.g., dual-core, quad-core, hexa-core, or anysuitable number of cores) or supercomputer. In some embodiments,processing circuitry is distributed across multiple separate processorsor processing units, for example, multiple of the same type ofprocessing units (e.g., two Intel Core i7 processors) or multipledifferent processors (e.g., an Intel Core i5 processor and an Intel Corei7 processor). In some embodiments, control circuitry 304 executesinstructions for an augmented content generation application stored inmemory (i.e., storage circuitry 308). Specifically, control circuitry304 may be instructed by the augmented content generation application toperform the functions discussed above and below.

In client/server-based embodiments, control circuitry 304 includescommunication circuitry suitable for communicating with an augmentedcontent generation application server or other networks or servers. Theinstructions for carrying out the above-mentioned functionality may bestored on a server. The communication circuitry may include a cablemodem, an integrated services digital network (ISDN) modem, a digitalsubscriber line (DSL) modem, a telephone modem, Ethernet card, or awireless modem for communication with other equipment, or any othersuitable communication circuitry. Such communication may involve theInternet or any other suitable communication networks or paths. Inaddition, communication circuitry may include circuitry that enablespeer-to-peer communication of devices, or communication of devices inlocations remote from each other.

Memory may be an electronic storage device provided as storage circuitry308 that is part of control circuitry 304. As referred to herein, thephrase “electronic storage device” or “storage device” should beunderstood to mean any device for storing electronic data, computersoftware, or firmware, such as random-access memory, hard drives,optical drives, solid state devices, quantum storage devices, or anyother suitable fixed or removable storage devices, and/or anycombination of the same. Nonvolatile memory may also be used (e.g., tolaunch a boot-up routine and other instructions). Cloud-based storage,described in relation to FIG. 2, may be used to supplement storagecircuitry 308 or instead of storage circuitry 308.

Control circuitry 304 may include image-generating circuitry and/orimage-processing circuitry. Control circuitry 304 may be used by userdevice 300 to receive and to display, to play, and/or to record visualcontent and/or augmented content. The circuitry described herein,including, for example, the image-generating circuitry orimage-processing circuitry, may be implemented using software running onone or more general purpose or specialized processors. If storagecircuitry 308 is provided as a separate device from user device 300, theimage-generating circuitry or image-processing circuitry may beassociated with storage circuitry 308.

Server device 320 may be implemented in a single, array, and/or cloudarchitecture. Server device 320 includes an I/O path 302 connected tocontrol circuitry 304 including processing circuitry 306 and storagecircuitry 308. I/O path 302, control circuitry 304, processing circuitry306, and storage circuitry 308 of server device 320 may be similar toI/O path 302, control circuitry 304, processing circuitry 306, andstorage circuitry 308 of user device 300, and may have similar featuresand functionality.

A user may send instructions to control circuitry 304 using a user inputinterface 310 of user device 300. User input interface 310 may be anysuitable user interface, such as a remote control, mouse, trackball,keypad, keyboard, touchscreen, touchpad, stylus input, joystick, voicerecognition interface, or other user input interfaces. Display 312 maybe a touchscreen or touch-sensitive display. In such circumstances, userinput interface 310 may be integrated with or combined with display 312.Speakers 314 may be provided as integrated with other elements of userdevice 300. User device 300 may further include one or more lenses 316 aand/or 316 b. Lenses 316 a, 316 b may be used to capture visual content,as described herein.

The augmented content generation application may be implemented usingany suitable architecture. For example, it may be a stand-aloneapplication wholly implemented on user device 300. The augmented contentgeneration application and/or any instructions for performing any of theembodiments described herein may be encoded on computer-readable media.Computer-readable media includes any media capable of storing data. Insome embodiments, the content generation application is aclient/server-based application. Data for use by a thick or thin clientimplemented on user device 300 is retrieved on demand by issuingrequests to a server, such as server device 320, remote from user device300. For example, user device 300 may receive inputs from the user viathe input interface 310 and transmit those inputs to the remote serverfor processing and generating the corresponding outputs. The generatedoutput is then transmitted to the user device 300 for presentation tothe user. Those skilled in the art will appreciate that AR device 102 ofFIG. 1 and AR device 260 of FIG. 2B may be implemented as user device300 of FIG. 3.

FIG. 4 is a flowchart of an illustrative process for augmenting visualcontent with a sponsored object, in accordance with some embodiments ofthe disclosure. A process 400 for providing augmenting visual contentwith a sponsored object may begin at block 402, where control circuitry,such as control circuitry 304 described above, captures visual content104 including a real-world view 250. Control circuitry 304 may capturevisual content 104 via one or more lenses 316. In some embodiments,control circuitry 304 retrieves visual content 104 from storage 308. Forexample, control circuitry 304 may capture an image of a person.

At block 404, control circuitry 304 receives an input selecting ageneric object with which to augment visual content 104. Controlcircuitry 304 may receive the input via user input interface 310. Forexample, control circuitry 304 receives a touch input from the user viauser input interface 310 selecting the generic object with which toaugment visual content 104 or receives a voice command selecting thegeneric object with which to augment visual content 104. In someembodiments, display 312 displays a list of available generic objects,and control circuitry 304 then receives a selection via user inputinterface 310 of one of the available generic objects.

At block 406, control circuitry 304 identifies a property of theselected object. Control circuitry 304 may analyze metadata associatedwith the selected object to identify the property of the selectedobject. In some embodiments, control circuitry 304 processes audioreceived as part of a voice command to identify the property of theselected object. The property of the selected object may be an explicitproperty (e.g., the voice command may specify the property of theselected object) or an implicit property (e.g., control circuitry 304first needs to identify what the selected object is in order to identifythe property of the selected object). For example, control circuitry 304may receive a voice command from the user, determine that the voicecommand includes an instruction to augment visual content 104 withaviator glasses, and then identify a property of aviator glasses (e.g.,a keyword, identifier, characteristic, etc.).

At block 408, control circuitry 304 determines whether the property ofthe selected object matches a property of a sponsored object 108. Asfurther described below with reference to FIG. 6, control circuitry 304may query a database 212 of sponsored objects for a sponsored objectmatching the property of the selected object identified at block 406. Inresponse to determining that the property of the selected object matchesa property of a sponsored object 108, the process 400 proceeds to block410. In response to determining that the property of the selected objectdoes not match a property of any sponsored objects, the process 400proceeds to block 412.

At block 410, control circuitry 304 augments visual content 104 withsponsored object 108 instead of the selected object. As describedfurther below with reference to FIGS. 5 and 7-9, control circuitry 304may overlay sponsored object 108 onto visual content 104, or maygenerate augmented content including sponsored object 108 and visualcontent 104. For example, control circuitry 304 may overlay sponsoredaviator glasses onto the image of the person.

At block 412, control circuitry 304 augments the visual content with theselected object. For example, control circuitry 304 overlays the genericobject selected at block 404 onto visual content 104.

FIG. 5 is a flowchart of a detailed illustrative process for augmentingvisual content 104 with a sponsored object 108, in accordance with someembodiments of the disclosure. A process 500 for augmenting visualcontent may begin at block 502.

At block 504, control circuitry 304 receives a second input indicating atarget position for the selected object. Control circuitry 304 mayreceive the second input via user input interface 310, such as via atouch input from the user specifying a position in visual content 104 inwhich to place the selected object. In some embodiments, controlcircuitry 304 receives the second input as a voice command, eithertogether with or separate from the first voice command selecting theobject, specifying where to place the selected object. For example,control circuitry 304 receives a voice command instructing controlcircuitry 304 to “show a sports car in the driveway.” Control circuitry304 may then process the voice command to determine that the sports caris the selected object and the driveway is the target position.

At block 506, control circuitry 304 determines whether visual content104 is being generated for display. Control circuitry 304 may determinewhether visual content 104 captured at block 402 is being generated fordisplay (e.g., visual content 104 is displayed on AR device 102), or wasmerely captured for processing purposes (e.g., visual content 104 is notdisplayed, that is, real-world view 250 is viewed through AR device260). In response to determining that visual content 104 is not beinggenerated for display, the process 500 proceeds to block 508. Inresponse to determining that visual content 104 is being generated fordisplay, the process 500 proceeds to block 510.

At block 508, control circuitry 304 generates for display sponsoredobject 108 at a position in real-world view 250 corresponding to thetarget position. For example, control circuitry 304 generates fordisplay the sponsored sports car such that it appears to be in thedriveway. In that regard, the target position, once identified, may be afixed position in real-world view 250, and control circuitry 304determines where on the display of AR device 260 to display sponsoredobject 108 such that it appears to be at the target position inreal-world view 250.

At block 510, control circuitry 304 generates for display augmentedvisual content wherein the sponsored object is overlaid onto real-worldview 250 at a position in real-world view 250 corresponding to thetarget position. For example, control circuitry 304 identifies aposition on visual content 104 matching the target position, andoverlays sponsored object 108 onto visual content 104 at the positionmatching the target position.

FIG. 6 is a flowchart of a detailed illustrative process for determiningwhether a property of a selected object matches a property of asponsored object 108, in accordance with some embodiments of thedisclosure. A process 600 for determining whether a property of aselected object matches a property of a sponsored object 108 may beginat block 602.

At block 604, control circuitry 304 queries a database 212 of sponsoredobjects 108 for a sponsored object 108 matching the property of theselected object identified at block 406. In some embodiments, controlcircuitry 304 sends a request for a sponsored object 108 matching theproperty of the selected object identified at block 406 to the contentserver 210, and the content server 210 queries database 212 forsponsored objects 108 matching the identified property.

At block 606, control circuitry 304 determines whether database 212 hasreturned a sponsored object 108 matching property of the selected objectidentified at block 406. For example, control circuitry 304 determineswhether any objects were returned in response to the query of block 604.In response to determining that database 212 has not returned anysponsored objects 108, the process 600 ends. In response to determiningthat database 212 has returned a sponsored object 108, the process 600proceeds to block 608.

At block 608, control circuitry 304 retrieves a template of sponsoredobject 108 matching the property of the selected object identified atblock 406. Control circuitry 304 may retrieve the template of sponsoredobject 108 from content source 210 and/or from database 212. Forexample, database 212 may return an identifier of a sponsored object 108in response to the query of block 604, and control circuitry 304 thenretrieves a sponsored object 108 matching the identifier from contentsource 210.

At block 610, control circuitry 304 generates an instance of sponsoredobject 108 based on the template retrieved at block 608. For example,the template may include design specifications and/or other datadescribing sponsored object 108, and control circuitry 304 may generatethe instance of sponsored object 108 according to the designspecifications.

At block 612, control circuitry 304 generates for display the instanceof sponsored object 108 generated at block 610. For example, controlcircuitry 304 may cause AR device 102 and/or AR device 260 to displaythe instance of sponsored object 108, as described at blocks 410, 508,and/or 510.

FIG. 7 is a flowchart of a detailed illustrative process for augmentingvisual content 104 with a sponsored object 108, in accordance with someembodiments of the disclosure. A process 700 for augmenting visualcontent may begin at block 702.

At block 704, control circuitry 304 identifies a subject object 106 invisual content 104. For example, control circuitry 304 processes visualcontent 104, such as by using object detection and/or patternrecognition algorithms, to identify a subject object 106 in visualcontent 104. In some embodiments, subject object 106 is a focus ofvisual content 104. For example, in an embodiment where visual content104 is an image of a person, the person is subject object 106 in visualcontent 104. In another example where the visual content is a real-worldview 250 of a house with a driveway, the driveway may be subject object106. Subject object 106 may be specified in input received from a useror may be automatically determined based on visual content 104. Forexample, if the user provides a command instructing AR device 260 to“show a sports car in the driveway,” control circuitry 304 may determinethat the user's specification of the driveway indicates subject object106 in visual content 104, and may then process visual content 104 toidentify the driveway.

At block 706, control circuitry 304 identifies an anchor point onsubject object 106. For example, control circuitry 304 determineswhether subject object 106 includes any identifiable landmarks. Inembodiments where subject object 106 is a person, the landmarks may befacial features, such as the person's eyes or nose, and controlcircuitry 304 may identify the person's eyes or nose in visual content104. Control circuitry 304 may then identify one or more of the facialfeatures as an anchor point. In some embodiments, control circuitry 304determines the landmarks to identify based on a type of subject object106 identified at block 704. For example, if control circuitry 304identifies subject object 106 as being a person, control circuitry 304may query a database for potential landmarks and/or templates oflandmarks that may be included in an image of a person, and then processvisual content 104 to identify the potential landmarks in visual content104, and therefrom the anchor points.

At block 708, control circuitry 304 correlates the anchor point onsubject object 106 with an anchor point on sponsored object 108. Forexample, control circuitry 304 may retrieve metadata associated withsponsored object 108 to identify anchor points on sponsored object 108.The metadata may further indicate which anchor points on sponsoredobject 108 correspond to which types of anchor points on subject object106. For example, the metadata may indicate that a particular anchorpoint on sponsored object 108 should be correlated with an anchor pointon the bridge of a person's nose. Control circuitry 304 then determineswhich anchor points on sponsored object 108 has been identified, andwhether those anchor points match any of the anchor points indicated inthe metadata.

At block 710, control circuitry 304 determines whether there are anyadditional anchor points on subject object 106 to be identified. Forexample, control circuitry 304 may determine, based on the type of thesubject object identified at block 704, whether there are any knownpotential landmarks remaining to be identified, and that, if so, thereare thus additional anchor points associated with those landmarks to beidentified. In response to determining that there are additional anchorpoints on subject object 106 to be identified, the process 700 returnsto block 706. In response to determining that there are no additionalanchor points on subject object 106 to be identified, the process 700proceeds to block 712.

At block 712, control circuitry 304 generates for display sponsoredobject 108 such that the anchor points on sponsored object 108identified at block 708 overlap with the anchor points on subject object106 identified at block 706. For example, control circuitry 304 overlayssponsored object 108 onto visual content 104 such that the anchor pointson sponsored object 108 overlap with the anchor points on subject object106.

FIG. 8 is a flowchart of a detailed illustrative process for augmentingvisual content 104 with a sponsored object 108, in accordance with someembodiments of the disclosure. A process 800 for augmenting visualcontent may begin at block 802.

At block 804, control circuitry 304 identifies subject object 106 in thevisual content 104. The process for identifying subject object 106 maybe similar to the process described at block 704, and will not berepeated for the purpose of brevity.

At block 806, control circuitry 304 identifies a feature of subjectobject 106. In some embodiments, after identifying subject object 106 invisual content 104, control circuitry 304 retrieves data indicatingpotential features of subject object 106. For example, if controlcircuitry 304 identifies subject object 106 as being a person, controlcircuitry 304 may retrieve data from content source 210 indicating thetypes of features found on a person. The features of subject object 106may be the head of subject object 106 or some other part indicative of arelative size of subject object 106. Control circuitry 304 may thenidentify the features in the visual content.

At block 808, control circuitry 304 determines whether there areadditional features to be identified. For example, control circuitry 304may determine whether each of the potential features of subject object106 retrieved from content source 210 have been identified or determinedto be unidentifiable in visual content 104. In response to determiningthat there are additional features of subject object 106 to beidentified, the process 800 returns to block 806. In response todetermining that there are no additional features of subject object 106left to be identified, the process 800 proceeds to block 810.

At block 810, control circuitry 304 determines a size of sponsoredobject 108. For example, control circuitry 304 may determine, based onthe size of a feature of subject object 106 identified at block 806,what the relative size of sponsored object 108 should be. In someembodiments, control circuitry 304 retrieves data (e.g., from themetadata associated with sponsored object 108) indicating the relativesize of sponsored object 108 to the feature of the subject object 106.

At block 812, control circuitry 304 generates for display sponsoredobject 108 according to the size determined at block 810. For example,control circuitry 304 generates an instance of sponsored object 108based on a template retrieved from content source 210, and scales theinstance according to the data indicating the relative size of sponsoredobject 108 to the feature of subject object 106.

FIG. 9 is a flowchart of a detailed illustrative process for augmentingvisual content 104 with a sponsored object 108, in accordance with someembodiments of the disclosure. A process 900 for augmenting visualcontent may begin at block 902.

At block 904, control circuitry 304 determines a position of a lightsource illuminating visual content 104. For example, control circuitry304 processes visual content 104 to identify shadows or areas ofrelatively high luminosity and areas of relatively low luminosity.Control circuitry 304 then determines an angle of the shadows, ordetermines a relative position of the areas of high luminosity to theareas of low luminosity, and determines, based on the angle of theshadows, or the relative position of the areas of high luminosity to theareas of low luminosity, a position of a light source illuminatingvisual content 104.

At block 906, control circuitry 304 determines whether there areadditional light sources illuminating visual content 104. For example,control circuitry 304 determines whether the angle of the shadows and/orrelative position of the areas of high luminosity to the areas of lowluminosity indicate that an unidentified light source is illuminatingvisual content 104. In response to determining that there are additionallight sources illuminating visual content 104, the process 900 returnsto block 904. In response to determining that there are no additionallight sources illuminating visual content 104, the process 900 proceedsto block 908.

At block 908, control circuitry 304 generates an instance of sponsoredobject 108. For example, as described at block 610, control circuitry304 retrieves a template of sponsored object 108 and generates fordisplay the instance of sponsored object 108 based on designspecifications included in the template.

At block 910, control circuitry 304 modifies the instance of sponsoredobject 108 based on the position of the light source determined at block904. For example, control circuitry 304 may adjust the color,brightness, luminosity, and/or shadows of the instance of sponsoredobject 108 based on the determined position of the light source.

At block 912, control circuitry 304 generates for display the modifiedinstance of sponsored object 108. For example, as described at block612, control circuitry 304 overlays the modified instance of sponsoredobject 108 onto visual content 104.

It should be noted that processes 400-900 or any step thereof could beperformed on, or provided by, any of the devices shown in FIGS. 1-3. Forexample, the processes may be executed by control circuitry 304 (FIG. 3)as instructed by an augmented content generation application implementedon an AR device 102, content source 210, and/or application server 220.In addition, one or more steps of a process may be omitted, modified,and/or incorporated into or combined with one or more steps of any otherprocess or embodiment (e.g., steps from process 600 may be combined withsteps from process 800). In addition, the steps and descriptionsdescribed in relation to FIGS. 4-9 may be done in alternative orders orin parallel to further the purposes of this disclosure. For example,each of these steps may be performed in any order or in parallel orsubstantially simultaneously to reduce lag or increase the speed of thesystem or method.

It will be apparent to those of ordinary skill in the art that methodsinvolved in the present invention may be embodied in a computer programproduct that includes a computer-usable and/or -readable medium. Forexample, such a computer-usable medium may consist of a read-only memorydevice, such as a CD-ROM disk or conventional ROM device, or arandom-access memory, such as a hard drive device or a computerdiskette, having a computer-readable program code stored thereon. Itshould also be understood that methods, techniques, and processesinvolved in the present disclosure may be executed using processingcircuitry.

The processes discussed above are intended to be illustrative and notlimiting. More generally, the above disclosure is meant to be exemplaryand not limiting. Only the claims that follow are meant to set bounds asto what the present invention includes. Furthermore, it should be notedthat the features and limitations described in any one embodiment may beapplied to any other embodiment herein, and flowcharts or examplesrelating to one embodiment may be combined with any other embodiment ina suitable manner, done in different orders, or done in parallel. Inaddition, the systems and methods described herein may be performed inreal time. It should also be noted, the systems and/or methods describedabove may be applied to, or used in accordance with, other systemsand/or methods.

What is claimed is:
 1. A method for augmenting visual content, themethod comprising: receiving a user input selecting a virtual object foraugmenting visual content including a real-world view; determiningwhether a property of the selected virtual object matches a property ofa sponsored object included in a database of sponsored objects; and inresponse to determining that the property of the selected virtual objectmatches the property of the sponsored object, augmenting the visualcontent including the real-world view with the sponsored object.
 2. Themethod of claim 1, further comprising in response to determining thatthe property of the selected virtual object does not match the propertyof the sponsored object included in the database of sponsored objects,augmenting the visual content including the real-world view with theselected object.
 3. The method of claim 1, wherein the user input is afirst input, and the method further comprises: capturing, via at leastone lens, the visual content; and receiving a second input indicating atarget position for the selected virtual object, wherein augmenting thevisual content including the real-world view with the sponsored objectcomprises generating, for display, the sponsored object at a position inthe real-world view corresponding to the target position.
 4. The methodof claim 1, wherein the user input is a first input, and the methodfurther comprises: capturing, via at least one lens, the visual content;and receiving a second input indicating a target position for theselected virtual object, wherein augmenting the visual content includingthe real-world view with the sponsored object comprises generating, fordisplay, augmented visual content, wherein the sponsored object isoverlaid onto the real-world view at a position in the real-world viewcorresponding to the target position.
 5. The method of claim 1, whereindetermining whether the property of the selected virtual object matchesthe property of the sponsored object included in the database ofsponsored objects comprises: identifying the property of the selectedvirtual object; and querying the database of sponsored objects for thesponsored object matching the property of the selected virtual object.6. The method of claim 1, wherein augmenting the visual contentincluding the real-world view with the sponsored object comprises:retrieving, from the database of sponsored objects, a template of thesponsored object; generating an instance of the sponsored object basedon the template of the sponsored object; and generating, for display,the instance of the sponsored object.
 7. The method of claim 1, whereinaugmenting the visual content including the real-world view with thesponsored object comprises: identifying a subject object in the visualcontent; identifying an anchor point on the subject object in the visualcontent; and generating, for display, the sponsored object based on theanchor point.
 8. The method of claim 7, wherein generating, for display,the sponsored object based on the anchor point comprises: correlatingthe anchor point on the subject object with an anchor point on thesponsored object; and generating, for display, the sponsored object suchthat the anchor point on the sponsored object overlaps the anchor pointon the subject object.
 9. The method of claim 1, wherein augmenting thevisual content including the real-world view with the sponsored objectcomprises: identifying a subject object in the visual content;identifying a feature of the subject object; determining, based on thefeature of the subject object, a size of the sponsored object; andgenerating, for display, the sponsored object according to thedetermined size.
 10. The method of claim 1, wherein augmenting thevisual content including the real-world view with the sponsored objectcomprises: determining a position of a light source illuminating thevisual content; generating an instance of the sponsored object;modifying the instance of the sponsored object by adjusting a visualcharacteristic of the sponsored object based on the determined positionof the light source illuminating the visual content; and generating, fordisplay, the modified instance of the sponsored object.
 11. A system foraugmenting visual content, the system comprising: control circuitryconfigured to: receive a user input selecting a virtual object foraugmenting the visual content including a real-world view; determinewhether a property of the selected virtual object matches a property ofa sponsored object included in a database of sponsored objects; and inresponse to determining that the property of the selected virtual objectmatches the property of the sponsored object, augmenting the visualcontent including the real-world view with the sponsored object.
 12. Thesystem of claim 11, wherein the control circuitry is further configuredto, in response to determining that the property of the selected virtualobject does not match the property of the sponsored object included inthe database of sponsored objects, augment the visual content includingthe real-world view with the selected object.
 13. The system of claim11, wherein the user input is a first input, and the control circuitryis further configured to: capture, via at least one lens, the visualcontent; receive a second input indicating a target position for theselected virtual object; and when augmenting the visual contentincluding the real-world view with the sponsored object, generate, fordisplay, the sponsored object at a position in the real-world viewcorresponding to the target position.
 14. The system of claim 11,wherein the user input is a first input, and the control circuitry isfurther configured to: capture, via at least one lens, the visualcontent; receive a second input indicating a target position for theselected virtual object; and when augmenting the visual contentincluding the real-world view with the sponsored object, generate, fordisplay, augmented visual content, wherein the sponsored object isoverlaid onto the real-world view at a position in the real-world viewcorresponding to the target position.
 15. The system of claim 11,wherein, when determining whether the property of the selected virtualobject matches the property of the sponsored object included in thedatabase of sponsored objects, the control circuitry is furtherconfigured to: identify the property of the selected virtual object; andquery the database of sponsored objects for the sponsored objectmatching the property of the selected virtual object.
 16. The system ofclaim 11, wherein, when augmenting the visual content including thereal-world view with the sponsored object, the control circuitry isfurther configured to: retrieve, from the database of sponsored objects,a template of the sponsored object; generate an instance of thesponsored object based on the template of the sponsored object; andgenerate, for display, the instance of the sponsored object.
 17. Thesystem of claim 11, wherein, when augmenting the visual contentincluding the real-world view with the sponsored object, the controlcircuitry is further configured to: identify a subject object in thevisual content; identify an anchor point on the subject object in thevisual content; and generate, for display, the sponsored object based onthe anchor point.
 18. The system of claim 17, wherein, when generating,for display, the sponsored object based on the anchor point, the controlcircuitry is further configured to: correlate the anchor point on thesubject object with an anchor point on the sponsored object; andgenerate, for display, the sponsored object such that the anchor pointon the sponsored object overlaps the anchor point on the subject object.19. The system of claim 11, wherein, when augmenting the visual contentincluding the real-world view with the sponsored object, the controlcircuitry is further configured to: identify a subject object in thevisual content; identify a feature of the subject object; determine,based on the feature of the subject object, a size of the sponsoredobject; and generate, for display, the sponsored object according to thedetermined size.
 20. The system of claim 11, wherein, when augmentingthe visual content including the real-world view with the sponsoredobject, the control circuitry is further configured to: determine aposition of alight source illuminating the visual content; generate aninstance of the sponsored object; modify the instance of the sponsoredobject by adjusting a visual characteristic of the sponsored objectbased on the determined position of the light source illuminating thevisual content; and generate, for display, the modified instance of thesponsored object.